Bringing Autonomic, Self-Regenerative Technology into Large Data Centers

With the introduction of blade servers and the emergence of techniques for spreading applications over clusters to exploit scalability, we’re suddenly seeing the emergence of enormous data centers. Google is said to operate more than 100,000 computers in its centers; Amazon has grown from perhaps 25 machines to as many as 2500 over just a few years, and similar stories are now common. These kinds of experiences presage a broader move towards very large cluster-style data centers in a broad range of commercial and military settings. Yet the technology for automated management of big systems is lagging. The most common distributed computing platforms require excessive care and feeding as a deployment grows, and are prone to instability and even melt-downs when large configurations are put under unexpected stress. At Cornell, the QuickSilver project is exploring selforganizing and self-repairing peer-to-peer technologies based on a new kind of epidemic communication protocol. The approach offers promise for such uses as managing, monitoring, and controlling very large-scale data centers. Moreover, same kinds of solutions may be useful in sensor networks.